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VIII. Claims 

Appendix A Claims 

Appendix B Evidence 

Appendix C Related Proceedings 

I. REAL PARTY IN INTEREST 

The real party in interest for this appeal is: 

Mitsubishi Denki Kabushiki Kaisha 

II. RELATED APPEALS AND INTERFERENCES 

There are no other appeals, interferences, or judicial proceedings which will directly 
affect or be directly affected by or have a bearing on the Board's decision in this appeal. 

III. STATUS OF CLAIMS 

A. Total Number of Claims in Application 
There are 8 claims pending in application. 

B. Current Status of Claims 

1. Claims canceled: 1-5, 10-35, and 38-40 

2. Claims withdrawn from consideration but not canceled: None 

3. Claims pending: 6-9, 36, 37, 41, and 42 

4. Claims allowed: None 

5. Claims rejected: 6-9, 36, 37, and 41 

C. Claims On Appeal 

The claims on appeal are claims 6-9, 36, 37, and 41 
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IV. STATUS OF AMENDMENTS 

Applicant did not file an Amendment After Final Rejection. However, after the non-final 
Office action issued on July 10, 2008, Applicant filed an Amendment on January 9, 2009, 
amending claims 6 and 7, canceling claims 10, 33-35 and 38-40, and adding new claim 42. 

V. SUMMARY OF CLAIMED SUBJECT MATTER 

The present invention relates to searching for an image or sequence of images (still or 
video image) containing a two-dimensional projection of a specific three-dimensional object of 
interest to the searcher. This is done by searching through stored representations of images or 
sequences of images. The stored representations represent each image or sequence of images 
using object descriptors for three-dimensional objects in the image or sequence of images. Each 
object descriptor includes a plurality of perspective view descriptors (associated together to form 
an object descriptor) which represent different two-dimensional perspective views of the three- 
dimensional object. The object descriptor includes a view descriptor of the object as it appears 
in the image (one two-dimensional perspective view) and view descriptors of the object in other 
two-dimensional perspective views (not appearing in the image or sequence of images). See for 
example, the different views of the 3-D cylinder object illustrated in Figure 2 of the 
Specification. A search is initiated by inputting a query, for example, in the form of an outUne 
which is the outline of the shape of a two-dimensional projection of the specific three- 
dimensional object of interest to the searcher. An object descriptor for the query object is 
derived and then compared with the stored object descriptors. More specifically, the query 
descriptor is compared with each of the plurality of perspective view descriptors in the stored 
object descriptor. If the query descriptor matches any of the perspective view descriptors, the 
overall object descriptor is considered a match and the corresponding image or sequence of 
images is retrieved, even if the perspective view of the object in the query does not match the 
perspective view of the object in the image or sequence of images. 

Independent claim 6 defines a method of searching for a query object in an image or 
sequence of images by processing signals corresponding to the images using a processor. 
(Specification p. 1, lines 4-5 and Fig. 1) The method includes, inter alia, providing a plurality of 
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stored images or sequences of images, each image or sequence of images containing a two- 
dimensional projection of a three-dimensional object in a perspective view; (Specification p. 5, 
lines 16-17 and Fig. 1) for each image or sequence of images, providing an object descriptor, 
each object descriptor including a plurality of perspective view descriptors, each perspective 
view descriptor representing the outline of the shape of a two-dimensional projection of said 
three-dimensional object from a different perspective view of said three-dimensional object, 
including a perspective view not in the image or sequence of images; (Specification p. 4, line 8 
to p. 5, line 15, Fig. 2 and Fig. 3) inputting a query in the form of a query outline which is the 
outline of the shape of a two-dimensional projection of a three-dimensional query object in a 
perspective view; (Specification page 6, lines 11-16 and lines 19-22, and Fig. 5) deriving a query 
object descriptor of the query object using the query outline; (Specification page 6, lines 17-18 
and Fig.5) comparing said query object descriptor with at least one of said object descriptors; 
(Specification page 7, lines 3-21, page 8, lines 1-5, page 8, line 21 to page 9, line 11, and Figs. 3 
and 5) selecting and displaying at least one result, which is an image or sequence of images 
containing an object for which comparison between the object descriptor and the query object 
descriptor indicates a degree of similarity between the query object and said object, wherein said 
object matches the query object but the selected and displayed image or sequence of images does 
not include said object in the same perspective view as the perspective view of the query object 
in said query. (Fig. 5, Specification page 9, lines 12-19). 

VI. GROUNDS OF REJECTION TO BE REVIEWED ON APPEAL 

A. Claims 6-9, 36, 37, and 41 stand rejected under 35 U.S.C. § 103(a) as being 
unpatentable over Eakins et al., "Retrieval of trade mark images by shape feature-the ARTISAN 
project). May 22, 1996 ("Eakins") in view of U.S. Patent No. 6,801,641 B2 to Eraslan 
("Eraslan"), and further in view of Japanese Publication No. JP 06-215105 to Haruo et al. 
("Haruo"). 

VII. ARGUMENT 

The rejection of claims 6-9, 36, 37, and 41 as unpatentable over the combination of 
Eakins, Eraslan and Haruo is improper because the combination fails to disclose "providing an 
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object descriptor, each object descriptor including a plurality of perspective view descriptors, 
each perspective view descriptor representing the outline of the shape of a two-dimensional 
projection of said three-dimensional object from a different perspective view of said three- 
dimensional objection, including a perspective view not in the image or sequence of images" as 
recited in claim 6, from which claims 7-9, 36, 37, and 41 depend. 

As discussed in the Specification, various methods, e.g., chain coding, quad-tree, and 
curvature scale space representations, of representing shapes of objects in images are known in 
the art. In addition, methods of searching images using stored representations are known. 
However, these known methods have disadvantages. In most cases, an object appearing in an 
image or video sequence is a projection of a 3-dimensional real object onto a 2-dimensional 
plane. The 2-dimensional shape or outline of an object appearing in an image depends on factors 
such as the viewing position, viewing angle, and camera and optical system parameters. Objects 
will therefore have different outlines associated with different views. (See 3-D cylinder object 
illustrated in Fig. 2 of the Specification) However, the known methods of representing images 
treat these different outlines as different objects. As a result, if, for example, a user inputs a 
query based on a front view of an object A, and object A only appears in a video sequence from 
the rear and side view, known methods will not successfiiUy match the query view of object A 
with the stored images including object A from the different view. The present invention 
overcomes this disadvantage with known methods by providing a system which associates 
different perspective views of a three-dimensional object appearing in an image or sequence of 
images. 

Accordingly, independent claim 6 defines a method of searching for a query object in an 
image or sequence of images by processing signals corresponding to the images using a 
processor. The method includes, inter alia, for each image or sequence of images, providing an 
object descriptor, each object descriptor including a plurality of perspective view descriptors, 
each perspective view descriptor representing the outline of the shape of a two-dimensional 
projection of said three-dimensional object from a different perspective view of said three- 
dimensional object, including a perspective view not in the image or sequence of images; 
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inputting a query in the form of a query outline which is the outline of the shape of a two- 
dimensional projection of a three-dimensional query object in a perspective view. 

In rejecting claim 6, the Examiner asserts that "it would have been obvious to one [of] 
ordinary skill in the art at the time of the invention was made to modify Eakins' system to 
include the different view of three-dimensional stored object in the database as taught by Eraslan 
in order to provide a faster search and retrieval system." In addition, the Examiner asserts that 
"Haruo also discloses providing a plurality of stored image representations of tree-dimensional 
[sic] objects, each image representation being associated with an object descriptor, each object 
descriptor including a plurality [of] view descriptors, each view descriptor of representing the 
outline of the shape of a projection of each one of the three-dimensional objects firom a different 
perspective view of the three-dimensional object." Therefore, the Examiner concludes that 
because Haruo allegedly suggests an object having different shapes, it would have been obvious 
to "modify Eakins and Eraslan to include each view descriptor representing a shape of an object 
as disclosed in Haruo in order to allow the retrieval of an image using shape." The Examiner's 
assertions are unfounded for the following reasons. 

First, the Examiner's conclusion of obviousness is based on an incorrect reading of 
Eraslan. On page 8 of the Office action dated July 10, 2008 ("Action"), the Examiner asserts that 
"Eraslan teaches providing a plurality of stored image representations of tree-dimensional [sic] 
objects, each image representation being associated with an object descriptor, each object 
descriptor including a plurality [of] view descriptors, each view descriptor of [sic] representing 
the outline of the shape of a projection of each one of the three-dimensional objects from a 
different perspective view of the three-dimensional [object]." To support this assertion, the 
Examiner points to column 4, lines 2-8 and Fig. 8 of Eraslan. 

Although Fig. 8 of Eraslan illustrates different perspective views of a nose, Eraslan fails 
to disclose that each of these different perspective views of the nose attribute are stored as part of 
the object descriptor for the nose attribute or that a perspective view represent the outline of the 
shape of a two-dimensional projection of the nose from a different perspective. To the contrary. 
Fig. 8 of Eraslan illustrates the fact that with the face-attribute index (e.g., 01) and the specific 
face-feature surface code (142), a completely 3-D, color face-feature surface which can be 



6 



MKM/PLC/ 



Application No.: 09/559,415 



Docket No.: 1906-0128P 



rendered graphically in the frontal view, left-profile view, right-profile view and all possible 
angled views of observation and analysis and be complied. See column 4, lines 35-60 and Figs 
7-13. Although Eraslan discloses that these 3-D surface maps can be used to analyze two- 
dimensional mug shots, nowhere in Eraslan is there any disclosure or suggestion of indexing 
outlines of the shape of a two-dimensional projection of the three dimensional facial feature as 
claimed. Therefore, even if, arguendo, one where to equate the shape codes of Eraslan as being 
equivalent to the claimed view descriptors, the shape codes of Eraslan do not represent different 
outiines of the shape of a two-dimensional projection of the perspective view of the same 3-D 
object, but rather entire 3-D surface maps for different objects in the facial feature family. 
Accordingly, the Examiner's assertion that Eraslan provides object descriptors as claimed in 
unfounded. 

Second, the Examiner's assertion that Haruo also discloses providing object descriptors 
as claimed in incorrect. Although Haruo discloses a method for creating two-dimensional 
projections from three-dimensional geometric shape information, nowhere in Haruo is there any 
disclosure or suggestion of storing perspective view descriptors representing the outiine of the 
shape of two-dimensional projection of a three-dimensional object in an image or sequence of 
images as claimed. At best, Haruo merely discloses a system for creating projections, not for 
creating view descriptors representing the outiine of the shape of a two-dimensional projection as 
claimed. Therefore, even if one skilled in the art were motivated to combine Eraslan and Haruo, 
the combination would merely provide a system for creating geometric shape information for 
numerous facial features and a system for creating two-dimensional projections of the three- 
dimensional facial surfaces once constructed. However, the combination would still fail to teach 
providing perspective view descriptors as claimed. 

Finally, Although Eakins discloses a system for searching for and retrieving similar 
images based in part on similarity between shape descriptors. The shape descriptors of Eakins 
are not equivalent to the claimed object or view descriptors. First, the shape descriptors of 
Eakins are not representative of three-dimensional objects within the stored images. To the 
contrary, the shape descriptors of Eakins are representative of an entire image, not a two- 
dimensional projection of a three-dimensional object as claimed. Second, the shape descriptors 
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of Eakins are based on individual boundaries or families of boundaries extracted from a stored 
image (i.e., the boundary exists in the image), not different perspective views of a three- 
dimensional object which are not in the image. 

Since Eakins, Eraslan, and Haruo each fail to disclose or suggest a method of searching 
for a query object in an image or sequence of images that includes, inter alia, providing an object 
descriptor including a plurality of perspective view descriptors, each perspective view descriptor 
representing the outline of the shape of a two-dimensional projection of a three-dimensional 
object from a different perspective view as claimed, the combination of these three references 
cannot possibly disclose or suggest said feature. Therefore, even if one skilled in the art were 
motivated to combine Eakins, Eraslan, and Haruo, which Appellant does not concede, the 
combination would still fail to render claim 6 unpatentable because the combination fails to 
disclose each and every claimed element. 

Claims 7-9, 36, 37, and 41 variously depend from independent claim 6. Therefore, 
claims 7-9, 36, 37, and 41 are patentable over the combination of Eakins, Eraslan, and Haruo for 
at least those reasons presented above with respect to claim 6. 

VIII. CLAIMS 

A copy of the claims involved in the present appeal is attached hereto as Appendix A. As 
indicated above, the claims in Appendix A include the amendments filed by Applicant on 
January 9, 2009. 

Dated: March 9, 2009 Respectfully submitted. 
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APPENDIX A 

Claims Involved in the Appeal of Application Serial No. 09/559,415 

Claim 6. A method of searching for a query object in an image or sequence of images by 
processing signals corresponding to the images using a processor, the method comprising: 

providing a plurality of stored images or sequences of images, each image or sequence of 
images containing a two-dimensional projection of a three-dimensional object in a perspective 
view; 

for each image or sequence of images, providing an object descriptor, each object 
descriptor including a plurality of perspective view descriptors, each perspective view descriptor 
representing the outline of the shape of a two-dimensional projection of said three-dimensional 
object from a different perspective view of said three-dimensional object, including a perspective 
view not in the image or sequence of images; 

inputting a query in the form of a query outline which is the outline of the shape of a 
two-dimensional projection of a three-dimensional query object in a perspective view; 

deriving a query object descriptor of the query object using the query outline; 

comparing said query object descriptor with at least one of said object descriptors; 

selecting and displaying at least one result, which is an image or sequence of images 
containing an object for which comparison between the object descriptor and the query object 
descriptor indicates a degree of similarity between the query object and said object, wherein said 
object matches the query object but the selected and displayed image or sequence of images does 
not include said object in the same perspective view as the perspective view of the query object 
in said query. 

Claim 7. A method as claimed in claim 6 wherein a query is input in the form of two or 
more two-dimensional outlines of an object, and wherein a query view descriptor is derived for 
each of said outlines, and wherein the step of comparing comprises comparing each of said query 
view descriptors with each view descriptor in each stored object descriptor to derive a plurality 
of view-similarity values. 
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Claim 8. A method as claimed in claim 7 wherein the view-similarity values are analyzed 
to derive object similarity values. 

Claim 9. A method as claimed in claim 6, wherein at least some of the object descriptors 
include view-independent descriptors which are related to shape or size of the object, and 
wherein the method comprises inputting a view-independent query value and the step of 
comparing compares the query value with the view-independent descriptors for the stored object 
descriptors. 

Claim 36. The method of claim 6, further comprising: 
deriving an object descriptor for an object in an image by: 

deriving a view descriptor of a first outline of a three-dimensional object in the image, 
deriving at least one additional view descriptor of the outline of the object in a different 
perspective view fi-om the perspective view in the image, and 

associating the two or more view descriptors to form the object descriptor. 

Claim 37. The method of claim 6, wherein said selecting and displaying includes 
selecting and displaying an image representation of an object having a different perspective view 
from perspective view of said query object based on said query object matching with at least two 
view descriptors including a view descriptor not representing perspective view of the object in 
the image representation. 

Claim 41. The method of claim 6, wherein each said view descriptor is a different 
representation of the object from a different perspective view of the three-dimensional object. 

Claim 42. The method of claim 6, wherein the step of comparing comprises comparing 
said query object descriptor with each of the plurality of perspective view descriptors. 
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APPENDIX B 

No evidence pursuant to §§ 1.130, 1.131, or 1.132 or entered by or relied upon by the 
examiner is being submitted. 
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APPENDIX C 

No related proceedings are referenced in II. above, hence copies of decisions in related 
proceedings are not provided. 
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